Papers with language identifiers
Language Identification and Named Entity Recognition in Hinglish Code Mixed Tweets (P18-3)
Copied to clipboard
| Challenge: | Named Entity Recognition (NER) is an important text analysis task . code-mixing occurs when lexical items and grammatical features from two languages appear in one sentence . |
| Approach: | They propose to use language identifiers, parts-of-speech tags and chunkers to analyze code-mixed data. |
| Outcome: | The proposed method outperforms the best baseline by 33.18%. |
Collecting Code-Switched Data from Social Media (L18-1)
Copied to clipboard
| Challenge: | a new method to identify code-switched data from the web is needed . code-witching is defined as the tendency of bilinguals to switch between languages . |
| Approach: | They propose a method that automatically collects code-switched tweets from the web . they use crowd-sourcing to obtain language identifiers for a subset of 8,000 tweets . |
| Outcome: | The proposed method identifies tweets as code-switched in languages L1 and L2 . it is compared to a Spanish-English corpus of code-witched tweets . |
FastSpell: The LangId Magic Spell (2024.lrec-main)
Copied to clipboard
| Challenge: | Language identification is a crucial component in the automated production of language resources. |
| Approach: | They propose a language identifier that combines fastText and Hunspell to give a second opinion before deciding which language to assign to a text. |
| Outcome: | The proposed language identifier is based on a pre-trained language identifier and a spell checker. |